Skip to content

Fixes 32792: let CreateTableRequest carry certification directly - #32793

Open
zak-nuccio wants to merge 6 commits into
open-metadata:mainfrom
zak-nuccio:feature/create-table-certification
Open

Fixes 32792: let CreateTableRequest carry certification directly#32793
zak-nuccio wants to merge 6 commits into
open-metadata:mainfrom
zak-nuccio:feature/create-table-certification

Conversation

@zak-nuccio

Copy link
Copy Markdown
Contributor

Describe your changes:

Fixes #32792

CreateTableRequest had no certification field, so database metadata ingestion had to create/update a table first and follow up with a separate JSON Patch to /certification. That patch is invisible to sourceHash, so a source-system change that is only a certification update produces the same hash as the previous run and the bulk fast-path silently skips it.

This adds an optional certification (AssetCertification) field to CreateTableRequest, threads it through TableMapper, and narrows the bot-overwrite guard in EntityRepository.updateCertification so a bot's explicit certification value is applied instead of always being reverted.

Type of change:

  • Improvement

High-level design:

EntityRepository.updateCertification() had a guard that reverted any bot PUT once a certification existed, regardless of what the request contained — a workaround for CreateTableRequest never carrying a certification value, so every bot PUT implicitly meant "no certification" and would otherwise have wiped it on every re-sync.

Now that a bot can supply an explicit value, the guard only needs to protect against omission:

if (operation.isPut()
    && !nullOrEmpty(original.getCertification())
    && updatedByBot()
    && !overrideMetadata
    && updatedCertification == null) {
  updated.setCertification(original.getCertification());
  return;
}

This mirrors the overrideMetadata escape hatch already used for description/owners/domains on the same class, rather than introducing a new mechanism:

  • Certification omitted from the request -> preserved (most connectors still won't populate it).
  • Certification explicitly supplied (same or different value) -> proceeds to the existing validate/apply/recordChange path.
  • overrideMetadata=true -> bypasses the guard entirely, consistent with the other protected fields.

No change was needed to generate_source_hash(): it hashes the full request via model_dump(), so certification is automatically included in the hash once it's a model field.

Alternatives considered:

  • Excluding certification from sourceHash as a special case — rejected, since a certification-only source change would still look "unchanged" to the bulk fast path.
  • Keeping the create-then-patch workaround but making the patch more reliable — rejected, doesn't fix the sourceHash blind spot.

Tests:

Use cases covered

  • Ingestion sends CreateTableRequest.certification on table create -> certification is applied.
  • Ingestion omits certification on a later run -> existing certification (e.g. set through the UI) is preserved, not wiped.
  • Ingestion sends a changed certification value on a later run -> the new value is applied and sourceHash differs, so the bulk fast-path doesn't skip it.
  • overrideMetadata=true still clears certification when the request omits it, matching existing behaviour for other protected fields.

Unit tests

  • I added unit tests for the new/changed logic.
  • Files added/updated:
    • openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/EntityRepositoryCertificationTest.java — 4 new tests exercising updateCertification()'s guard directly (bot-omit-preserves, bot-explicit-applies, bot-omit-with-overrideMetadata-clears, human-omit-clears).
    • ingestion/tests/unit/utils/test_source_hash.py — certification-added/changed/omitted/equivalent-payload hash stability tests for generate_source_hash.

Backend integration tests

  • Not applicable (no new API endpoint; existing PUT /tables already accepts CreateTableRequest).

Ingestion integration tests

  • Not applicable (schema/model change only; connectors adopting this field are a separate follow-up).

Manual testing performed

  1. Ran the full EntityRepositoryCertificationTest suite (32/32 passing) and the full org.openmetadata.service.jdbi3.** package (616/616 passing).
  2. Ran ingestion/tests/unit/utils/test_source_hash.py and ingestion/tests/unit/topology/test_runner.py (all passing) after regenerating Python models from the updated schema.
  3. Verified mvn spotless:apply and mvn spotless:check clean on openmetadata-service.

UI screen recording / screenshots:

Not applicable.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed — certification is a new optional field on CreateTableRequest, so no migration is needed (existing stored data is unaffected).
  • I have added tests (unit / integration / Playwright as applicable) and listed them above.
  • I have added tests around the new logic.

CreateTableRequest had no certification field, so database metadata
ingestion had to create/update a table first and follow up with a
separate JSON Patch to /certification once the entity existed. That
patch is also invisible to sourceHash, so a source system change that
is only a certification update produces the same hash as the previous
run and the bulk fast-path skips it, silently dropping the update.

- createTable.json: add optional certification (AssetCertification).
- TableMapper: pass CreateTable.certification through to the entity.
- EntityRepository.updateCertification: the bot-overwrite guard used
  to revert any bot PUT once a certification existed, regardless of
  what the request contained (a workaround for CreateTableRequest
  never carrying a value). Narrow it to only preserve certification
  when the request omits it, matching the overrideMetadata escape
  hatch already used for description/owners/domains on the same
  class. An explicit certification value from a bot is now applied
  instead of reverted.
- generate_source_hash needs no change: it hashes the full request via
  model_dump(), so certification is automatically included once it is
  a model field.

Tests: source-hash stability/change coverage in test_source_hash.py,
and four EntityRepositoryCertificationTest cases exercising the guard
directly (omit-preserves, explicit-applies, overrideMetadata-clears,
non-bot-clears).
Copilot AI lite review requested due to automatic review settings September 7, 2026 03:59
@zak-nuccio
zak-nuccio requested review from a team as code owners September 7, 2026 03:59
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Comment thread ingestion/tests/unit/utils/test_source_hash.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The schema change is not reflected in committed UI-generated TypeScript types (e.g., generated CreateTable/CreateTableRequest models still lack certification), leaving generated artifacts out of sync.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR enables database ingestion to set/update table certification in the same PUT /tables request by adding certification to the CreateTable request model, ensuring certification-only changes affect sourceHash and are not skipped by the bulk fast-path.

Changes:

  • Add optional certification (AssetCertification) to the CreateTableRequest JSON schema.
  • Thread certification into table entity creation via TableMapper.
  • Refine EntityRepository.EntityUpdater.updateCertification() to preserve existing certification only when a bot PUT omits the field (and overrideMetadata=false), while allowing explicit bot-provided certification values.
  • Add Java unit tests for the updated certification guard and Python unit tests asserting sourceHash changes with certification payload changes.
File summaries
File Description
openmetadata-spec/src/main/resources/json/schema/api/data/createTable.json Adds optional certification field to the CreateTable request schema.
openmetadata-service/src/main/java/org/openmetadata/service/resources/databases/TableMapper.java Maps request certification onto the Table entity during create/update mapping.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java Narrows bot PUT “preserve certification” guard to omission-only + honors overrideMetadata.
openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/EntityRepositoryCertificationTest.java Adds unit tests covering the new guard behavior (bot omit/explicit, overrideMetadata, human).
ingestion/tests/unit/utils/test_source_hash.py Adds tests verifying generate_source_hash() changes/stability with certification added/changed/omitted.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 112 to 119
"type": "string",
"minLength": 1,
"maxLength": 32
},
"certification": {
"description": "Certification for a table",
"$ref": "../../type/assetCertification.json"
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the generated TS models were stale — fixed in 56d4e5c by regenerating via json2ts-generate-all.sh (same as the TypeScript Type Generation CI workflow runs). Only createTable.ts and bulkCreateTable.ts changed, matching the schema edit.

The backend always recomputes AssetCertification.appliedDate/expiryDate
server-side from AssetCertificationSettings when a certification is
applied, ignoring whatever the request sent for those two fields. Only
tagLabel reflects a real change. Hashing appliedDate/expiryDate would
destabilize sourceHash on every ingestion run if a connector ever
populates them with a run-time-relative value, defeating the bulk
fast-path this PR's certification support relies on.

Scoped to the certification field only (not a global key strip),
since expiryDate is also a legitimate, meaningful field on regular tag
metadata (TagLabelMetadata) that should stay part of the hash.
Copilot AI review requested due to automatic review settings September 7, 2026 04:23
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The CreateTable schema change needs the corresponding committed UI-generated TypeScript schema outputs regenerated/updated to keep spec-derived clients in sync.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

openmetadata-spec/src/main/resources/json/schema/api/data/createTable.json:119

  • The CreateTable JSON schema now includes the new optional certification field, but the committed UI-generated TypeScript types still define CreateTable without certification. This will leave UI/client type generation out of sync with the spec; please regenerate and commit the updated TS schema outputs under openmetadata-ui/src/main/resources/ui/src/generated/ (e.g., .../api/data/createTable.ts).
    },
    "certification": {
      "description": "Certification for a table",
      "$ref": "../../type/assetCertification.json"
    }

ingestion/tests/unit/utils/test_source_hash.py:567

  • This line exceeds the repo's Ruff line-length (120) and will likely fail formatting/lint checks; wrap the AssetCertification(...) call across multiple lines.
            certification=AssetCertification(tagLabel=tag_label, appliedDate=1800000000000, expiryDate=1999999999999),
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

name="test_table",
databaseSchema="service.db.schema",
columns=[Column(name="id", dataType=DataType.INT)],
certification=AssetCertification(tagLabel=tag_label, appliedDate=1700000000000, expiryDate=1731536000000),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked — both lines are 118 characters, under the 120 limit (awk '{print length}' confirms it), and ruff check/ruff format --check both pass clean on this file as committed. Not making a change here; flagging as a false positive rather than splitting these two calls across more lines.

createTable.ts and bulkCreateTable.ts were out of sync with the
certification field added to createTable.json. Regenerated via
json2ts-generate-all.sh, matching the TypeScript Type Generation CI
workflow.
Copilot AI review requested due to automatic review settings September 7, 2026 05:19
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The newly added ingestion unit test includes lines that are likely to violate the repository’s Ruff line-length=120 constraint and can fail CI linting.

Review details

Suppressed comments (2)

ingestion/tests/unit/utils/test_source_hash.py:561

  • This line is likely to exceed the repository's Ruff 120-character line-length limit, which can fail CI linting for ingestion tests. Please wrap the AssetCertification construction across multiple lines (similar to the _certification helper) to keep lines <= 120 chars.
            certification=AssetCertification(tagLabel=tag_label, appliedDate=1700000000000, expiryDate=1731536000000),

ingestion/tests/unit/utils/test_source_hash.py:567

  • This line is likely to exceed the repository's Ruff 120-character line-length limit, which can fail CI linting for ingestion tests. Please wrap the AssetCertification construction across multiple lines to keep lines <= 120 chars.
            certification=AssetCertification(tagLabel=tag_label, appliedDate=1800000000000, expiryDate=1999999999999),
  • Files reviewed: 6/8 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

updateCertification()'s unchanged-check compared the full
AssetCertification object, including appliedDate/expiryDate. The
server always recomputes those two fields when a certification is
applied, so a request that legitimately can't know the server's
current dates (e.g. an ingestion connector re-sending the same
certification every run) would never compare equal - triggering a
spurious re-apply and version bump on every non-bulk PUT even when the
certification itself hadn't changed.

Compare by tagLabel.tagFQN instead, matching the identity check
applyCertification() already uses for its own idempotency.
Copilot AI review requested due to automatic review settings September 7, 2026 05:37
@zak-nuccio

Copy link
Copy Markdown
Contributor Author

Re: the version-churn edge case in Gitar's latest review — good catch, fixed in 1fd6ca5. updateCertification()'s unchanged-check compared the full AssetCertification object, so a request that couldn't know the server's recomputed appliedDate/expiryDate would never compare equal, re-applying and bumping the version on every non-bulk PUT. Now compares by tagLabel.tagFQN only, matching the identity check applyCertification() already uses for its own idempotency. Added updateCertificationSameTagLabelWithDifferentDatesIsNotReapplied to cover it.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

updateCertification() returns early when the certification tag is unchanged without restoring the stored certification, allowing request-supplied date fields to be persisted without a version bump/audit record.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/8 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment on lines +10065 to 10068
if (certificationTagUnchanged) {
LOG.debug("Certification unchanged");
return;
}
Comment on lines +258 to +261
invokeUpdateCertification(updater);

assertEquals(4000000000000L, updated.getCertification().getAppliedDate());
verify(tagUsageDAO, never())
The tagLabel-only unchanged check skipped re-applying the certification
but left the request's arbitrary appliedDate/expiryDate on the updated
entity, which the caller then persists regardless of whether this
method recorded a change. Restore the stored (server-authoritative)
certification, including its real dates, before returning, so the
client's date fields can never silently overwrite storage without a
version bump or audit record.
Copilot AI review requested due to automatic review settings September 7, 2026 05:48
@zak-nuccio

Copy link
Copy Markdown
Contributor Author

Good catch, fixed in 03c28a2. The tag-unchanged early return skipped re-applying the certification but left the request's own appliedDate/expiryDate on updated, which the caller persists regardless of whether this method recorded a change — so the client's dates could silently overwrite storage with no audit trail. Now restores updated.setCertification(origCertification) (the stored, server-authoritative value) before returning. Updated the regression test to assert both dates match the original stored values, not the request's.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are cohesive, well-tested across backend and ingestion hashing behavior, and the remaining feedback is a minor comment-clarity nit.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:10057

  • The comment says we compare "by tagLabel only", but the actual comparison is only on tagLabel.tagFQN (not the full TagLabel). Updating the comment will prevent confusion about what changes are considered "unchanged" (e.g., labelType/state differences are ignored).
  • Files reviewed: 6/8 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 7, 2026 05:56
@zak-nuccio

Copy link
Copy Markdown
Contributor Author

Fair, fixed in 499283a — comment now says tagLabel.tagFQN explicitly and notes that other TagLabel fields (labelType, state, etc.) are intentionally ignored for this comparison.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@gitar-bot

gitar-bot Bot commented Sep 7, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 2 resolved / 2 findings

Adds optional certification field to CreateTableRequest so database metadata ingestion can carry certification directly instead of requiring a follow-up patch. This fixes sourceHash instability when certification changes — appliedDate/expiryDate are now excluded from the hash, and bot-supplied values proceed through the validate/apply path instead of being reverted on each sync. Includes unit tests for the certification guard logic and sourceHash stability. No issues found.

✅ 2 resolved
Edge Case: Certification appliedDate/expiryDate can destabilize sourceHash

📄 ingestion/tests/unit/utils/test_source_hash.py:45-55
generate_source_hash hashes the full request and _remove_volatile_fields only strips href/deleted/inherited, so AssetCertification's appliedDate/expiryDate are included in the hash. If a connector ever populates these with run-time-relative or now() values, the sourceHash will differ on every ingestion run even when nothing changed, defeating the bulk fast-path (the very optimization this PR aims to preserve). The unit tests only pass because they hard-code fixed timestamps. Consider treating appliedDate/expiryDate as volatile (excluded from the hash) so only the certification tag value drives change detection.

Edge Case: Bot-supplied certification re-applies and bumps version each run

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:10053-10067
Now that a bot can supply certification via CreateTableRequest, a request carrying the same tagLabel each sync will still fail the Objects.equals(origCertification, updatedCertification) check at line 10053: the stored origCertification has server-computed appliedDate/expiryDate, while the request's updatedCertification has them null. This drives execution into the recompute/applyCertification/recordChange path every run, bumping the entity version even when nothing changed. The bulk fast-path masks this because sourceHash drops those dates and skips re-processing, but a non-bulk PUT /tables re-sync would churn versions. Consider comparing only the tagLabel (or nulling the request dates) before deciding the certification is unchanged.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow CreateTableRequest to carry certification directly

2 participants